> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/mlfoundations/open_clip/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Preparation

> Prepare training data in CSV and WebDataset formats for CLIP training

## Overview

OpenCLIP supports two main data formats for training:

1. **CSV Format**: Simple, good for small to medium datasets (\<10M samples)
2. **WebDataset Format**: Efficient, required for large-scale datasets (>10M samples)

This guide covers how to prepare, format, and optimize your training data.

## CSV Format

CSV format is the simplest way to specify image-text pairs for training.

### Basic CSV Structure

Create a CSV or TSV file with image paths and corresponding captions:

```csv theme={null}
filepath,title
/data/images/img001.jpg,"A photo of a cat sitting on a windowsill"
/data/images/img002.jpg,"A dog playing fetch in the park"
/data/images/img003.jpg,"Sunset over the ocean with sailboats"
```

### CSV Parameters

Control CSV parsing with these flags:

```bash theme={null}
python -m open_clip_train.main \
    --train-data "/path/to/train_data.csv" \
    --dataset-type csv \
    --csv-separator "\t" \
    --csv-img-key filepath \
    --csv-caption-key title \
    # ... other arguments
```

**Parameters:**

* `--csv-separator`: Column delimiter (default: `\t` for tab)
* `--csv-img-key`: Column name for image paths (default: `filepath`)
* `--csv-caption-key`: Column name for captions (default: `title`)

### CSV Format Examples

**Tab-separated (default):**

```tsv theme={null}
filepath\ttitle
/data/img1.jpg\tA cat
/data/img2.jpg\tA dog
```

**Comma-separated:**

```csv theme={null}
image_path,caption
/data/img1.jpg,"A cat playing with yarn"
/data/img2.jpg,"A dog in the snow"
```

```bash theme={null}
--csv-separator "," \
--csv-img-key image_path \
--csv-caption-key caption
```

**Custom columns:**

```csv theme={null}
id,url,description,tags
1,/img/001.png,"Red sports car",vehicle
2,/img/002.png,"Mountain landscape",nature
```

```bash theme={null}
--csv-img-key url \
--csv-caption-key description
```

### CSV Best Practices

<Check>
  **Good practices:**

  * Use absolute paths for image files
  * Quote captions containing commas or special characters
  * Keep CSV files on fast storage (SSD)
  * Split large CSVs into train/val sets
</Check>

<Warning>
  **Limitations:**

  * CSV format is slow for datasets >10M samples
  * Random access is inefficient for shuffling
  * No built-in data sharding for distributed training
  * **Recommendation:** Use WebDataset for datasets >10M samples
</Warning>

### Validation Data

Provide a separate CSV for validation:

```bash theme={null}
python -m open_clip_train.main \
    --train-data "/data/train.csv" \
    --val-data "/data/val.csv" \
    --dataset-type csv \
    # ... other arguments
```

## WebDataset Format

WebDataset is a streaming dataset format optimized for large-scale training. It stores data in `.tar` archives with efficient sequential access.

### WebDataset Structure

A WebDataset consists of multiple `.tar` files, each containing paired image and text files:

```
cc12m-train-0000.tar
├── 000000.jpg
├── 000000.txt
├── 000001.jpg
├── 000001.txt
├── 000002.jpg
└── 000002.txt

cc12m-train-0001.tar
├── 001000.jpg
├── 001000.txt
└── ...
```

**Key points:**

* Each sample consists of an image file and a text file with the same name
* Files are grouped into `.tar` archives (typically 1,000-10,000 samples each)
* Archives can be stored locally or accessed remotely (S3, HTTP)

### Creating WebDatasets with img2dataset

[img2dataset](https://github.com/rom1504/img2dataset) is the recommended tool for converting image-text datasets to WebDataset format.

#### Installation

```bash theme={null}
pip install img2dataset
```

#### Basic Usage

**Input:** Parquet or CSV file with image URLs and captions

```python theme={null}
# input.parquet
url,caption
https://example.com/img1.jpg,"A cat"
https://example.com/img2.jpg,"A dog"
```

**Convert to WebDataset:**

```bash theme={null}
img2dataset --url_list input.parquet \
    --input_format "parquet" \
    --url_col "url" \
    --caption_col "caption" \
    --output_folder dataset \
    --output_format webdataset \
    --processes_count 16 \
    --thread_count 64 \
    --image_size 256 \
    --resize_mode "shortest" \
    --resize_only_if_bigger True \
    --encode_format "jpg" \
    --encode_quality 95
```

**Output:**

```
dataset/
├── 00000.tar
├── 00001.tar
├── 00002.tar
└── ...
```

#### Conceptual Captions 3M Example

See the [CC3M img2dataset example](https://github.com/rom1504/img2dataset/blob/main/dataset_examples/cc3m.md) for a complete walkthrough:

```bash theme={null}
# Download CC3M metadata
wget https://ai.google.com/research/ConceptualCaptions/download

# Convert to WebDataset
img2dataset --url_list cc3m.tsv \
    --input_format "tsv" \
    --url_col "url" \
    --caption_col "caption" \
    --output_folder cc3m_webdataset \
    --output_format webdataset \
    --processes_count 16 \
    --thread_count 64 \
    --image_size 224 \
    --resize_mode "center_crop"
```

### Training with WebDataset

Use the `--dataset-type webdataset` flag:

```bash theme={null}
python -m open_clip_train.main \
    --train-data "/data/cc12m/cc12m-train-{0000..2175}.tar" \
    --train-num-samples 10968539 \
    --dataset-type webdataset \
    --batch-size 256 \
    # ... other arguments
```

**Key parameters:**

* `--train-data`: Path with glob pattern for .tar files
* `--train-num-samples`: Total number of samples (required)
* `--dataset-type webdataset`: Specify WebDataset format

### Glob Patterns for WebDataset

WebDataset supports bash-style glob patterns:

```bash theme={null}
# Range of files: 0000 to 2175
--train-data "/data/train-{0000..2175}.tar"

# Range with padding: 00000 to 41455
--train-data "/data/train-{00000..41455}.tar"

# Multiple directories
--train-data "/data/shard1/train-{0000..1000}.tar" \
             "/data/shard2/train-{0000..1000}.tar"
```

### Dataset Resampling

For large datasets, enable sampling with replacement:

```bash theme={null}
python -m open_clip_train.main \
    --train-data "/data/laion400m/{00000..41455}.tar" \
    --train-num-samples 400000000 \
    --dataset-type webdataset \
    --dataset-resampled \
    # ... other arguments
```

**Benefits of `--dataset-resampled`:**

* Enables efficient epoch-based training on streaming datasets
* Allows training for fewer than one full epoch (via `--train-num-samples`)
* Required when using multiple data sources with upsampling

### Partial Epochs for Large Datasets

For very large datasets (LAION-2B), train on a fraction of an epoch:

```bash theme={null}
python -m open_clip_train.main \
    --train-data "/data/laion2b/{00000..100000}.tar" \
    --train-num-samples 135646078 \
    --dataset-type webdataset \
    --dataset-resampled \
    --epochs 32 \
    # ... other arguments
# train-num-samples = 2.17B / 16 ≈ 135M (1/16 of full dataset per "epoch")
# More frequent checkpoints and evaluation
```

## Multiple Data Sources

Combine multiple datasets using the `::` separator:

```bash theme={null}
python -m open_clip_train.main \
    --train-data "/data/cc12m/train-{0000..2175}.tar::/data/laion400m/{00000..41455}.tar" \
    --dataset-type webdataset \
    --dataset-resampled \
    # ... other arguments
```

### Data Upsampling Factors

By default, samples from each source are proportional to dataset size. Use `--train-data-upsampling-factors` to control weighting:

```bash theme={null}
# Equal sampling from both datasets
python -m open_clip_train.main \
    --train-data "/data/cc12m/train.tar::/data/cc3m/train.tar" \
    --train-data-upsampling-factors "1::4" \
    --dataset-resampled
    # ... other arguments
# CC12M: ~12M samples, weight 1
# CC3M: ~3M samples, weight 4
# Effective sampling ratio: 12M:12M (equal)
```

**Examples:**

```bash theme={null}
# Default behavior (proportional to size)
--train-data "dataset_a.tar::dataset_b.tar"
# Dataset A: 1000 samples, Dataset B: 100 samples
# Sampling probability: A=90%, B=10%

# Equal sampling
--train-data "dataset_a.tar::dataset_b.tar" \
--train-data-upsampling-factors "0.001::0.01"
# or equivalently: --train-data-upsampling-factors "1::10"
# Sampling probability: A=50%, B=50%

# Custom weighting
--train-data "high_quality.tar::medium_quality.tar::low_quality.tar" \
--train-data-upsampling-factors "2::1::0.5"
# High quality: 2x weight, Medium: 1x, Low: 0.5x
```

### Combining LAION and CC12M Example

```bash theme={null}
python -m open_clip_train.main \
    --train-data "/data/cc12m/cc12m-{0000..2175}.tar::/data/laion400m/{00000..41455}.tar" \
    --train-data-upsampling-factors "1::1" \
    --dataset-type webdataset \
    --dataset-resampled \
    --batch-size 256 \
    --model ViT-B-32
# Both datasets sampled equally despite different sizes
```

## Remote Data Loading

WebDataset supports loading data from remote sources:

### AWS S3

```bash theme={null}
python -m open_clip_train.main \
    --train-data "pipe:aws s3 cp s3://my-bucket/train/train-{0000..2175}.tar -" \
    --dataset-type webdataset \
    # ... other arguments
```

### HTTP/HTTPS

```bash theme={null}
python -m open_clip_train.main \
    --train-data "https://example.com/datasets/train-{0000..2175}.tar" \
    --dataset-type webdataset \
    # ... other arguments
```

### Google Cloud Storage

```bash theme={null}
python -m open_clip_train.main \
    --train-data "pipe:gsutil cat gs://my-bucket/train-{0000..2175}.tar" \
    --dataset-type webdataset \
    # ... other arguments
```

<Note>
  Remote data loading requires network bandwidth. For best performance:

  * Use local storage when possible
  * Ensure high-bandwidth network connection
  * Monitor network utilization with `iftop` or `nethogs`
</Note>

## Data Format Specifications

### Image Formats

Supported image formats:

* JPEG (.jpg, .jpeg)
* PNG (.png)
* WebP (.webp)
* Any format supported by PIL/Pillow

**Recommendations:**

* Use JPEG for photographs (smaller file size)
* Use PNG for images with transparency or text
* Store at reasonable resolution (224-512px for most models)

### Text Formats

**WebDataset:** Plain text files (.txt)

```
A photo of a cat sitting on a windowsill
```

**CSV:** Quoted strings

```csv theme={null}
/path/to/image.jpg,"A photo of a cat sitting on a windowsill"
```

**Caption guidelines:**

* Keep captions concise but descriptive
* Typical length: 5-20 words
* Remove special characters that may cause parsing issues
* Use UTF-8 encoding for international text

## Data Quality Considerations

### Image Quality

<Check>
  **Good practices:**

  * Filter out corrupted or unreadable images
  * Remove duplicates
  * Ensure minimum resolution (e.g., 224x224)
  * Verify aspect ratios are reasonable
  * Check for NSFW content if needed
</Check>

### Caption Quality

<Check>
  **Good practices:**

  * Remove empty or very short captions (\<3 words)
  * Filter out captions with excessive special characters
  * Remove duplicate captions
  * Consider language filtering for multilingual datasets
  * Remove personally identifiable information (PII)
</Check>

### Dataset Size Recommendations

| Use Case             | Dataset Size | Format            | Example               |
| -------------------- | ------------ | ----------------- | --------------------- |
| Experimentation      | 1M-10M       | CSV or WebDataset | CC3M, CC12M           |
| Small-scale training | 10M-100M     | WebDataset        | YFCC15M               |
| Medium-scale         | 100M-500M    | WebDataset        | LAION-400M            |
| Large-scale          | 500M+        | WebDataset        | LAION-2B, DataComp-1B |

## Example Datasets

### Conceptual Captions 3M (CC3M)

```bash theme={null}
# Download and prepare CC3M
wget https://ai.google.com/research/ConceptualCaptions/download

# Convert to WebDataset using img2dataset
img2dataset --url_list Train_GCC-training.tsv \
    --input_format "tsv" \
    --url_col "url" \
    --caption_col "caption" \
    --output_folder cc3m \
    --output_format webdataset \
    --processes_count 16 \
    --thread_count 64

# Train
python -m open_clip_train.main \
    --train-data "cc3m/{00000..00331}.tar" \
    --train-num-samples 3000000 \
    --dataset-type webdataset \
    --model ViT-B-32
```

### Conceptual Captions 12M (CC12M)

```bash theme={null}
# Download CC12M
wget https://github.com/google-research-datasets/conceptual-12m/blob/master/cc12m.tsv

# Convert to WebDataset
img2dataset --url_list cc12m.tsv \
    --input_format "tsv" \
    --url_col "url" \
    --caption_col "caption" \
    --output_folder cc12m \
    --output_format webdataset \
    --processes_count 32 \
    --thread_count 128

# Train
python -m open_clip_train.main \
    --train-data "cc12m/cc12m-{0000..2175}.tar" \
    --train-num-samples 10968539 \
    --dataset-type webdataset \
    --dataset-resampled \
    --model ViT-B-32
```

### LAION Datasets

LAION datasets are available as pre-processed WebDatasets:

* [LAION-400M](https://laion.ai/blog/laion-400-open-dataset/)
* [LAION-2B](https://laion.ai/blog/laion-5b/)
* [LAION-5B](https://laion.ai/blog/laion-5b/)

See [LAION documentation](https://github.com/LAION-AI/laion-datasets) for download instructions.

## Preprocessing and Augmentation

### Image Preprocessing

Images are automatically preprocessed during training:

1. **Resize** to model input size (default: 224x224)
2. **Normalize** with ImageNet statistics
3. **Data augmentation** (optional)

Control preprocessing with these flags:

```bash theme={null}
--image-mean 0.485 0.456 0.406 \
--image-std 0.229 0.224 0.225 \
--image-interpolation bicubic \
--image-resize-mode shortest
```

### Data Augmentation

Custom augmentation via `--aug-cfg`:

```bash theme={null}
--aug-cfg scale_range=0.08::1.0 ratio_range=0.75::1.33
```

See [Configuration](/training/configuration) for all augmentation options.

## Storage Optimization

### Tar File Size

Recommended tar file sizes:

* **1,000-10,000 samples per .tar** for local storage
* **Smaller tars (1,000-5,000)** for remote/networked storage
* Balance between random access and I/O efficiency

### Compression

WebDataset .tar files can be compressed:

```bash theme={null}
# Create compressed tars
tar czf train-0000.tar.gz *.jpg *.txt

# Use in training (automatic decompression)
--train-data "train-{0000..1000}.tar.gz"
```

**Tradeoff:** Smaller storage vs. CPU overhead for decompression

### SSD vs HDD

| Storage  | Random Access | Sequential | Best For                            |
| -------- | ------------- | ---------- | ----------------------------------- |
| NVMe SSD | Excellent     | Excellent  | CSV, small WebDataset               |
| SATA SSD | Good          | Good       | CSV, WebDataset                     |
| HDD      | Poor          | Good       | Large WebDataset (sequential reads) |
| Network  | Variable      | Variable   | Remote WebDataset                   |

## Troubleshooting

### Slow Data Loading

**Symptom:** Low GPU utilization (\<80%)

**Solutions:**

1. Increase `--workers` (more data loading processes)
2. Use faster storage (SSD instead of HDD)
3. Convert CSV to WebDataset for large datasets
4. Reduce image resolution in preprocessing
5. Check network bandwidth for remote data

### Corrupted Images

**Symptom:** Training crashes with PIL errors

**Solutions:**

1. Filter corrupted images during preprocessing:
   ```bash theme={null}
   img2dataset --skip_reencode False
   ```
2. Add error handling in custom data pipeline
3. Validate all images before creating WebDataset

### Missing Files

**Symptom:** FileNotFoundError during training

**Solutions:**

1. Use absolute paths in CSV files
2. Verify file permissions
3. Check glob patterns for WebDataset
4. Ensure all nodes have access to shared storage (multi-node)

### Unbalanced Data Sources

**Symptom:** Model overfits to larger dataset

**Solutions:**

1. Use `--train-data-upsampling-factors` to balance sampling
2. Create weighted mixture of datasets
3. Train on larger dataset first, then fine-tune on smaller

## Data Validation Script

Validate your WebDataset before training:

```python theme={null}
import webdataset as wds
from PIL import Image
import io

# Test loading from WebDataset
dataset = wds.WebDataset("/data/train-0000.tar")
for i, sample in enumerate(dataset):
    if i >= 10:  # Check first 10 samples
        break
    
    # Verify image can be loaded
    img = Image.open(io.BytesIO(sample["jpg"]))
    print(f"Sample {i}: {img.size}, caption: {sample['txt'][:50]}...")
    
print("Validation complete!")
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Single-Node Training" icon="server" href="/training/single-node">
    Train on prepared data with single machine
  </Card>

  <Card title="Configuration" icon="sliders" href="/training/configuration">
    Configure training parameters and data augmentation
  </Card>

  <Card title="Distributed Training" icon="rocket" href="/training/distributed-training">
    Optimize data loading for distributed training
  </Card>

  <Card title="Training Overview" icon="book" href="/training/overview">
    Return to training overview
  </Card>
</CardGroup>